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ABSTRACT 

This report addresses three major issues concerning 
the Army's program for developing extension training materials: (1) 
indications of low usage of Army Training Extension Course lessons by 
soldiers in the field; (2) improvements needed in the process for 
developing extension training materials; and (3) the need for further 
evaluation before the Army commits itself to procuring and fielding a 
new videodisc electronic information delivery system programmed to 
cost approximately $127.3 million . The document begins with a letter 
from Frank C. Conahan, Director of the U. S. General Accounting 
Office (GAO) to John 0. Marsh, Jr., the Secretary of the Army, which 
provides a critique of the Army's Extension Training Program, 
recommends further evaluation before purchase is made of new 
videodisc equipment, and introduces the four appendices that make up 
the ma^or part of the document. Appendix I provides background 
information on extension training materials; the objectives, scope 
and methodology of this study; and major findings of the study. 
Appendix II consists of a letter from the Senior Associate Director 
of the U.S. General Accounting Office expressing concern over whether 
the acquisition plans for videodisc equij^ent are justified based on 
the low usage of current training materials by soldiers in the field. 
Appendix III contains a letter from the Principal Deputy Assistant 
Secretary of Defence in response to the GAO report "Observations on 
Army's Plan to Buy an Electronic Video-Disc Delivery System," also 
included in Appendix III. The final appendix consists of the GAO 
draft report, "Improvements Needed in the Army's Program for 
Developing Extension Training Materials for Use by Soldiers in Field 
Units," which contains five findings and four recommendations and the 
Department of Defense's reaction to each. A letter from the 
Department of Defense in response to the draft GAO report is also 
provided. (JB) 
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The Army has about 16,000 different training mate- 
rials—such as soidiars manuals arid audio-visual 
materials— and plam\ to add about 24;5d0 more train- 
ing items and several thousand videb^disc'systems 
to provide training in field units.. Because these 
materials will cost millions of dollars. GAO reviewed 
the Army process for determining its training mate- 
rial needs and its justification for the new video-disc 
system. 

GAO found that the Army had not established accept- 
able usage levels for its training materials, had not 
obtained field unit input on unit needs for training 
materials, and had not identified the usage and effec- 
tiveness of individual training materials. Also, the 
Army has not sufficiently demonstrated its need for 
the large numbers of video-disc equipment it plans 
to acquire for use in the field. GAO recommends that 
the Army (1) take several actions to improve its 
needs determination process for training materials 
and (2) further evaluate the planned video-disc sys- 
teni procurement. 
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The Honorable John 0. Marshy Jr. 
The Secretary of the Army 

Dear Mr. Secretary: 

We have reviewed the Army's program for developing extension 
training materials to improve individual soldier proficiency in 
field units. This report addresses (1) indications of low usage 
of Training Extension Course lessons by soldiers in the fields 
(2) improvements needed in the process for developing extension 
training materials ^ and (3) the need for further evaluation 
before the Army commits itself tc procuring and fielding a new 
video-disc electronic information delivery system programmed to 
cost about $127.3 million. 

As you kiowy the U.S. Army Training and Doctrine Command 
(TRADOC)r through its service schools^ develops training 
materials for (1) resident school training of soldiers and (2) 
support of unit and individual soldier training — commonly 
referred to as extension training. Each school determines the 
type^ quality^ and quantity of training materials within assigned 
military occupational specialties. TRADOC's U.S. Army Training 
Support Center is the overall program manager fot developing 
extension training materials. 

Currently^ ^bout 16^000 different materials^ such as 
soldiers manuals and audio-visual materials^ are available for 
use by soldiers in troop units. About 24^500 more are under 
development or are planned for future development at a cost of 
many millions of dollars. 

Costs to develop extension training materials are not 
identified and budgeted separately. For development of both 
resident and extension training materials^ TRADOC spent about 
$92 million and $105.7 million in fiscal years 1983 and 1984^ 
respectively. TRADOC estimated that the average cost to develop 
certain categories of materials we reviewed generally ranged from 
$2^000 for television audio-visual tapes to about $7^000 for each 
Training Extension Course lesson and $20^000 to $40^000 for 
development of each motion picture film. 
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ARMY STUDIES AND AUDITS INDICATE 

LOW USE OF TRAINING EXTENSION COURSE LESSONS 

Training Extension Course lessons are designed to teach 
specific tasksr such as assembling an M16 rifle^ operating a 
gasoline lantern^ and setting up a mobile kitchen trailer. Army 
studies^ audits^ and other data since 1979 have characterized 
usage as low for the Training Extension Course lessonji;. This low 
usage has precipitated questions about the need for these 
materials. However r the Army does not have criteria specif yinq 
what are acceptable usage levels for individual training 
materials or categories of materials. Furthermore^ tl:e r.rmy does 
not obtain routine feedback to reflect the usage levels for 
individual extension training materials or their ef f ectiveness. 

None of the studies ^ audits^ or surveys fully explored the 
reasons for what was characterized and reported as low usage 
levels. Some studies didr however ^ conclude that factors other 
than need for training or the training materials, such as limited 
command emphasis^ could impact usage levels. The indicated low 
usage of the materials cannot be attributed solely to or equated 
with a lack of need for training or for a particular type of 
training material^ such as extension training materials. But we 
believe questions about the causes for low usage and what is 
meant by low usage cannot be adequately addressed until criteria 
are available defining acceptable usage levels and data is 
systematically collected on individual training materials usage 
levels and the reasons for those levels. 

An ongoing Army survey of several thousand soldiers will 
again identify overall usage levels for the Training Extension 
Course program^ but it will not identify the usage level or 
effectiveness of individual lessons. 

PROCESS FOR DEVELOPING EXTENSION 
TRAINING MATERIALS NEEDS IMPROVEMENT 

The Army has initiated actions to improve its process for 
developing extension training materials by issuing guidance for 
their development and by making the above-mentioned materials 
usage survey. In addition^ the Department of Defense ^ on October 
2, 1984, in a response (app. Ill) to our July 11, 1984, report 
(app. II) on the planned procurement of new video-disc equipment, 
stated that lessons learned from various studies and surveys had 
been used to develop plans to improve program management. 
However, we believe TRADOC needs to further improve its method 
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for determining the need for extension training materials before 
it spends the millions of dollars planned for developing 
additional training materials. 

Program guidance issued by TRADOC states that the materials 
must be needed and wanted by field units. However r we found that 
the TRADOC schools are not obtaining field unit input on their 
needs and desires for extension training materials. The schools 
are developing the materials on the basis of their identification 
of critical tasks associated with a military occupational 
specialty and the application of a needs analysis procedure to 
each task. According to TRADOC's guidance, the key determina- 
tions to be made during the application of the needs analysis 
procedure inv'lude (1) v;hether the task can be taught sufficiently 
well without training materials^ (2) whether existing materials 
can satisfy t^ie need, and (3) what kind of product should be 
developed if uew materials are required. 

Although the needs analysis procedure in TRADOC 's guidance 
states that an extension training material should be developed 
only if it is required and meets the stated needs of field units^ 
the procedure does not specify how actual or stated needs are to 
be defined or how the schools are to determine field needs. We 
believe TRADOC nee-ds to define what the determination of field 
needs for traininrj products means and include in its guidance 
procedures for obtaining field unit input in determining the need 
for extension training materials. 

Given the indicated low une and the large number of training 
materials planned and under development ^ the Army needs to ensure 
that lessons learned from usage of extension training materials 
previously fielded are considered before proceeding with its 
planned large-scale Gffort to develop new materials^ many of 
which are for new or revised 'military occupational specialties 
related to system3 under development. In doing this^ the Army 
also needs to obtain user feedback on individual materials and to 
develop criteria which define acceptable extension training 
materials usage levels. Key indicators are whether the products 
are used and improve soldier proficiency. 

FURTHER EVALUATION NEEDED BEFORE ARMY BUYS 
AND FIELDS NEW VIDEO-DISC EQUIPMENT 

The Army plans to procure a new video-disc electronic 
information delivery system to replace existing paper and other 
audio-visual delivery systems. About $127.3 million has been 
programmed to procure this system and to develop and convert 
training materials to video-disc in fiscal years 1984-90. In 
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our July 11, 1984, report (app. II), we expressed our concerns 
about these planned expenditures because 

— over 80 percent of the 20,000 units of video--disc equip- 
ment in the approved acquisition objectives are tor use by 
soldiers in the field and 

— the justification for the purchases did not include an 
analysis of past extension training materials usage. 

The Department of Defense commented on our report (app. Ill) 
as follows: 

1. Army analysis of extension training material usage was 
continuous and would influence the fielding plan for the 
new video-disc system. 

2. Procurement of the video-disc hardware was scheduled 
first for Army schools and was not scheduled for field 
units until fiscal year 1987; the largest buy was 
scheduled for fiscal year 1990. 

3. Procurement would be cut, if the video-disc system had 
not proven itself. With respect to the systen; proving 
itself, DOD stated that pilot fielding of video-disc 
lessons to units indicated favorable acceptance and 
usage levels and that additional validation of the 
equipment woulJ occur in the school setting prior to 
fielding in fiscal year 1987, 

4. Usage was only one of several factors to consider in 
determining requirements for extension training. Other 
factors included the number and types of tasks taught 
during initial entry at training centers and schools 
compared with those which must be taught by the Active 
and Reserve component organizations to which individuals 
were assigned. 

5. Video-disc training materials would be distributed on 
the basis of user demand. In addition, the number and 
types of equipment would be based on factors such as 
expected usage of the training materials and scheduling 
of the skill qualification test for assigned personnel. 

We agree that Army analysis of extension training material 
should be continuous and that the actions planned to further 
evaluate the need for large numbers of the video-disc equipment 
are steps in the right direction. However, we believe that 
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the Army needs to proceed cautiously in procuring the new video- 
disc system for field use until after (1) it has field tested the 
new equipment and (2) it has made the needed management improve- 
ments in the program for developing extension training mate-- 
rials. We believe that the results of these improvements should 
be incorporated in the reevaluation of the need for large numbers 
of video-disc equipment because (1) criteria for acceptable usage 
levels have not yet been established, (2) adequate data is not 
now available on the use and effectiveness of extension training 
materials, and (3) plans to further evaluate user acceptance of 
the video-disc equipment in service schools where use is required 
will not, in our opinion, provide sufficient indications of how 
it will be used by soldiers in field units. According to the 
Army, only a small supply of video-disc equipment will become 
available before fiscal year 1989 for field use and evaluation. 
The first large-scale procurement is for fiscal year 1989. 

RECOMMENDATIONS 

We recommend that to improve the extension training 
materials program, you direct the Commander, U.S. Army Training 
and Doctrine CommaTRJv to 

— develop criteria for what is to be considered an actual or 
a stated field need for extension training materials to 
include what are acceptable usage levels for the 
materials, 

— delineate procedures in TRADOC guidance which specify (1) 
how schools should obtain field input on actual needs and 
(2) how this input is to be used in developing materials, 
and 

— obtain feedback on individual training materials usage and 
effectiveness for the purposes of incorporating lessons 
learned into the development of requirements for new 
extension training materials and determining the need to 
revise existing materials. 

Concerning the Army's plans to procure the new video*-disc 
equipment units for field use, we recommend that you reevaluate 
the fiscal year 1989 procurement plan using test results of field 
use and need. Tiiis evaluation should consider (1) the usage 
levels for extension training materials and the reasons for such 
usage and (2) the status of other management improvements needed 
in the program for developing such materials, such as the devel- 
opment of criteria defining acceptable usage levels and improve- 
ments in the criteria and procedures for determining field needs 
for extension training materials. 
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AGENCY COMMENTS AND OUR EVALUATION 

On May 16^ 1985^ DOD commented on the draft of this report 
(app. IV). DOD concurred in the recommendations concerning 
improvements needed in tne extension training materials program 
and stated that TRADOC would be tasked with 

— developing criteria for identifying field needs and 
defining acceptable usage levels; 

— reviev^ing and improving its procedures for training 

materials development to include procedures for obtaining 
field input on training materials needs and guidance on 
how this input would be used in developing materials; and 

— ensuring^ as part of this review, the systematic checking 
of training materials use and effectiveness and periodic- 
ally assessing the effectiveness of the feedback procedure 
in ensuring materials and program effectiveness. 

DOD also concurred in our recommendation on the need for 
ensuring that the Army's reevaluation of the planned procurement 
of new video-disc equipment units for field use include consid- 
eration of the usage levels for extension training materials and 
the status of other management improvements needed in the program 
for developing such materials. But DOD did not agree with our 
original proposal that the Army reevaluate the planned procure- 
ments for field units befora distributing the video-disc equip- 
ment to the field in fiscal year 1987. DOD commented that the 
proper time for assessing the Army's video-disc field support 
program would be fiscal year 1989 after a small supply of initial 
training materials and equipment would become available for field 
use and evaluation. DOD agreed to (1) limit the initial fiscal 
year 1986 video-disc equipment procurements for field units to 
the small quantity required to test actual field use and needs 
and (2) reevaluate the field needs for the equipment before the 
planned large-scale procurements in fiscal year 1989. 

We believe the actions taken and planned will bring about 
the desired improvements in the extension training materials 
program and help ensure that only those products needed are 
developed and retained. Also, we believe the Army's plans to 
field test a small number of the new video-disc equipment will 
help ensure that the system is procured only if it is demon- 
strated that the equipment is needed and will be effectively used 
by soldiers in the field. 
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Our findings are discussed in more detail in appendix I. 

As you know, 31 U.S.C. § 720 requires the head of a federal 
agency to submit a written statement on actions taken on our 
recommendations to the Senate Committee on Governmental Affairs 
and the House Committee on Government Operations not later than 
60 days after the date of the report and to the House and Senate 
Committees on Appropriations with the agency's first request for 
appropriations made more than 60 days after the date of the 



We are sending copies of this report to the Committees 

listed above; the House and Senate Committees on Armed Services; 

the Director, Office of Management and Budget; and the Secretary 
of Defense. 



report. 



Sincerely yours. 




Frank C. Conahan 
Director 
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IMPROVEMENTS NEEDED IN THE ARMY'S PROGRAM 
FOR DEVELOPING EXTENSION TRAINING MATERIALS 
FOR USE BY SOLDIFRS IN FIELD UNITS 

BA CKGROUND 

The U.S. Arrny Training and noctrine Command (TRAOOC) , 
through its service schools, develops training materials for (1) 
resident school training of soldiers and (2) support of unit and 
individual soldier training— commonly referred to as extension 
training. These schools determine the type, quality, and quan- 
tity of training materials. Each school has the responsibility 
within assigned military occupational specialties. For example, 
the transportation school is responsible for military occupa- 
tional specialties, such as truck drivers and watercraft 
operators. TRADOC's U.S. Army Training Support Center at Fort 
Eustis, Virginia, is the overall program manager for developing 
extension training materials. 

The primary purpose of extension training materials is 
supporting training in troop units in both Active and Reserve 
components. These materials include items such as training 
devices, correspondence courses, field manuals, soldier manuals. 
Training Extension Course (TEC) lessons, audio-visual materials 
(educational television tapes and motion picture films), graphic 
training aids (charts, cards, and other materials), and resident 
training materials provided to units. TEC lessons are designed 
to teach specific tasks, such as assembling and disassembling an 
M16A1 rifle, setting up a mobile kitchen trailer, and operating 
and maintaining a gasoline lantern. 

Development of extension training materials has been under 
way for many years. There are now about 16,000 different exten- 
sion training materials in inventory, and 24,500 more are under 
development or are planned for future development. 

Costs to develop extension training materials are not iden- 
tified separately from development of resident school training 
materials. TRAOOC spent about $92 million in fiscal year 1983 
and about $105.7 million in fiscal year 1984 for developing 
resident and extension training materials. According to informa- 
tion provided by TRADOC, the materials being produced in fiscal 



^Extension training is training performed in field units. 
Materials developed by Army schools and exported to the units 
for this purpose are referred to as extension training 
materials. 
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year 1983 were predominantly extension training materials. Also, 
information provided by TRAOOC showed that estimated costs to 
develop the categories of materials2 we reviewed generally range 
from $2,000 for television audio-visual tapes to about $^,000 for 
each TEC lesson. Costs to develop motion picture films average 
from $20,000 to $40,000 each. 

OBJECTIVES, SCOPE, AND METHODOLOGY 

Our primary objective was to evaluate the Army's system for 
determining requirements for extension training materials. 
Additionally, we reviewed the Army's justification for its olans 
to field a new video-disc electronic information delivery system. 

For our review of requirements for extension trainino 
materials, we selected three categories of extension training 
materials--TEC lessons, audio-visuals, and graphic training 
aids— because Army studies, audits, and other data indicated low 
usage of these materials by soldiers in field units. These 
categories represent over 50 percent of all extension training 
materials fielded and planned. 

In performing our fieldwork from January to September 1984, 
we visited Army Headquarters; TRADOC; and the following 7 of 24 
schools, which were selected at random and represent a cross 
section of TRADOC schools: 

Combat arms ; 

Infantry, Fort Benning, Georgia 

Air Defense Artillery, Fort Bliss, Texas 

Combat support ; 

Aviation, Fort Rucker, Alabama 
Signal, Fort Gordon, Georgia 

Combat service support ; ^ 

Quartermaster, Fort Lee, Virginia 



2Estimated development costs for graphic trainiig aids were not 
provided . 

3we originally selected two schools in each category, but becau 
of a recent organizational change which resulted in the additi 
of another school at Fort Eustis, our review was conducted at 
three combat service support schools. 
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Aviation Logisticsj. Port Eustis^ Virginia 
Transportation, Port Eustis, Virginia 

In our work on the system for determining requirements for 
extension training materials, we interviewed Army Headquarters 
and TRADOC personnel and reviewed regulations and records on 
guidance for determining need for developing extension training 
materials. At the schools, we interviewed personnel who 
determine the need for training materials and reviewed trip 
reports of visits to field installations and other documentation 
on the systems and procedures for determining need for extension 
training materials. 

We also obtained information on training materials usage 
from Army studies, audits, and other data reflecting training 
materials usage. Because TEC lessons represented a large portion 
of the total materials — about 30 percent of all fielded training 
materials and about 55 percent of those in the three categories 
reviewed — we limited our review of training materials usage to 
that available for these lessons. We did not verify the data and 
figures in the Army studies and audits reflecting usage of 
training materials. In addition, we visited the U.S. Army Porces 
Command Headquarters and two of its installations — Port Bragg, 
North Carolina, and Port Carson, Colorado — to interview 
commanders and enlisted personnel to obtain a user perspective on 
the training materials. We did not assess the quality or 
effectiveness of individual materials. 

Concerning the Army's proposed procurement of video-disc 
equipment, we interviewed personnel at Army Headquarters and 
TRADOC and reviewed the requirements documentation and other 
supporting records for the planned procurement. 

Our review was performed in accordance with generally 
accepted government audit standards. 

LOW USE OP TRAINING MATERIALS 
RAISES QUESTIONS ABOUT 
THE NEED POR THE MATERIALS 

Data is not routinely collected on extension training 
materials usage. However, the Army has expressed concern over 
the large numbers of extension training materials, and several 
Army studies and audits conducted since 1979 have classified 
usage as low for the TEC lessons. In 1980, there were about 
20,400 extension training materials. An Army effort to reduce 
the extension training materials inventory by 50 percent resulted 
in the elimination of 6,635 different items, or about 33 percent, 
of the then-existing materials, leaving an inventory of about 
14,000 materials. 
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A 1979 Army Research Institute study showed that (1) about 
50 percent of the 3^284 soldiers who responded to questionnaires 
had used TEC materials and (2) 82 percent of recorded uses for 
selected units during a 2-month test period were directed by unit 
commanders. This study concluded that some factors contributing 
to what is considered to be low usage may have been ignorance 
about the program by some soldiers^ a low level of command 
emphasis^ and the unavailability of the materials in the units. 
The study stated that these factors could have their greatest 
impact on the voluntary^ individual type of use for which the 
program was designed. It also concluded that this could explain 
why most identified uses were mandatory and were by groups. 

More recent Army studies and audits have also disclosed 
limited use of TEC lessons. For i-xample^ in 1982^ the Army Audit 
Agency reported that use of TEC lessons at the 5th Infantry 
Division^ Port Polk^ Louisiana^ was very low. It reported that 

— 52 of 90 soldiers interviewed had never used the materials 
and 21 had never heard of them; 

— 7 of 44 companies had included the materials in their 
training schedules^ but in these 7 companies^ the 
materials were scheduled for use on 36 of 441 available 
training days; and 

— the average number of soldiers using these materials each 
month was generally very low compared with the battalion 
strength level— from 5 to 22 percent for 5 battalions and 
42 percent for the sixth battalion reviewed. 

A July 1982 report based on a TRADOC Systems Analysis 
Activity study of training effectiveness for the new Hawk missile 
system found that while from 71 to 79 percent of 1,506 soldiers 
in five Hawk-missile-related military occupational specialties 
liked the TEC lessons, only 32 to 56 percent had actually 
completed them. As did other studies, this study concluded that 
extension training materials were viewed in a positive manner but 
the usage seemed low. 

The U.S. Army Training Support Center sponsored a 1982 
survey on TEC lesson usage. The survey was conducted by the 
Military Personnel Center. The survey showed that 47 percent of 
3,521 officers and 43 percent of 12,111 enlisted personnel 
surveyed had not used the lessons during the previous 12 months. 
Beccuse of the Center's concern over the indicated low usage 
levels, it initiated another survey of the usage of these lessons 
which included a statistical sample of several thousand personnel 



APPENDIX I 



APPENDIX I 



in the Active^ Reserve^ and National Guard components of the 
Army, Although the results of this survey were originally due in 
April 1984, the Army was continuing its review of the results at 
the time of our review. This survey will show overall indica- 
tions of usage of training lessons; it will not assess the use 
and effectiveness of individual lessons* 

While the indicated low usage levels can raise questions 
about the need for these extension training materials ^ such low 
use cannot be attributed solely to a lack of need for training or 
for certain types of training materials* None of the studies or 
audits specifically addressed the need for these extension train- 
ing lessons or the reason for the usage levels identified. They 
did, however r state that other factors^ such as soldier aware- 
ness, command emphasis^ and availability of materials^ can influ- 
ence the usage level* Nevertheless ^ we believe the consistent 
results of the studies^ surveys ^ and audits over the last 5 years 
showing low usage indicate the Army needs to (1) define what 
acceptable usage levels are^ (2) obtain feedback on usage levels 
and reasons for the usage , and (3) use this information as appro- 
priate to improve the method for deciding to develop materials 
and/or ensure necessary use by the soldiers. 

THE ARMY NEEDS TO IMPROVE THE PROCESS 

FOR DEVELOPING EXTENSION TRAINING MATERIALS 

TRADOC may be developing and revising extension training 
materials that are not needed or that may not be used suffi- 
ciently to justify their cost. More specif ically^ we found that 
the schools are not developing extension training materials on 
the basis of actual field needs as required in TRADOC 's December 
1982 guidance. Although the guidance discusses determining need 
for training support^ it does not specify how schools should 
obtain and use input on actual field needs. In addition^ the 
Army has not established an adequate feedback system for use in 
assessing the effectiveness and determining the usage level of 
extension training materials once they have been developed* 

In view of several factors — (1) the earlier attempt to 
eliminate half of the then-existing materials^ (2) the consistent 
indications of low usage of materials in the field based on the 
Army's evaluations^ and (3) the thousands of extension training 
materials under development and programmed for future 
development — we believe the Army needs to (1) specify how input 
on actual field training needs is to be obtained and used before 
materials are developed and (2) establish a system for obtaining 
information on their use and effectiveness. We believe that 
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unless the.'^e improvements are made, TRADOC will continue to 
develop and revise extension training materials for which there 
is little assurance of need. 

Decisions to develop extension 
training materials should consider 
actual field needs 

TRADOC s schools plan to spend millions of dollars 
developing thousands of extension training materials on the basis 
of a specified task analysis* The task analysis approach does 
not specifically query field units regarding their need for these 
materials, even though the December 1982 TRADOC guidance 
specifies that the materials must be needed and wanted by field 
units. 

TRADOC s December 1982 guidance states that (1) new 
requirements for training support must be determined only after 
careful consideration of the actual needs of field units, (2) the 
materials must be usable, and (3) particular emphasis must be 
placed on the ability of units to effectively use the materials 
and services. The guidance provides for a front-end analysis to 
be performed for each military occupational specialty which 
results in the identification of critical tasks to be taught in 
the unit. The guidance then provides for step-by-step 
procedures, called a needs analysis procedure, to be applied to 
each critical task to determine requirements for extension 
training materials. 

According to the guidance, the key determinations to be made 
during application of the needs analysis procedure to each 
critical task are (1) whether the task can be taught sufficiently 
well without training materials, (2) whether existing materials 
can satisfy the need, and (3) what kind of product should be 
developed if new materials are required. Although the needs 
analysis procedure states that an extension training material 
"should be developed only if it is required and meets the stated 
needs of field units," the procedure does not specify how the 
schools are to determine field needs* While it does address the 
use of evaluation exercises, such as the skill qualification 
test, 4 in the determination, the guidance does not define an 
actual or a stated field need. 



^A performance-oriented test used to measure individual 
proficiency in performing critical tasks related to a soldier's 
primary military occupational specialty. 
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Generally^ school personnel who develop training products 
told us they decide to develop extension training materials on 
the basis of the steps in the needs analysis procedure without 
obtaining field input on their training neads. They said that 
only in rare instances did they receive requests from field units 
for development of extension training materials. 

Personnel at the schools we visited identified two potential 
sources of field input on need for materials — periodic visits to 
field units by school personnel and the rotation of personnel 
from field units to the schools. However , except for those few 
instances when field units asked for training materials, the 
schools could not provide any documentation from the trip reports 
on their visits or otherwise showing where actual training needs 
of field units had been identified from these sources. 

Even though the school personnel stated that the front-end 
analysis is performed to identify critical tasks and that the 
needs analysis procedure is applied to each task, they said they 
automatically program for development all those identified 
critical tasks for which extension training materials are not 
already available. Furthermore, the TRC lessons are distributed 
to units on the basis of the assigned military occupational 
specialties through what the Army calls a "push" system, rather 
than determining requirements based on user demand. Audio 
visuals and graphic training aids are distributed to the 
approximately 136 Training and Audio Visual Support Centers 
worldwide where units and individuals can request those materials 
from the centers. 

Without field input on training needs, materials could be 
developed which are not needed or do not meet user needs. The 
Army needs to establish procedures Cor obtaining this input so 
the schools can consider field needs when making decisions on 
developir;g training materials for them. Using techniques such as 
the visits to field installations, the Army should specify how 
actual needs of field units are to be defined, identified, and 
satisfied . 

Improved program evaluation is 
needed for the extension training 
materials program 

Closely related to the need for procedures specifyinq how 
field needs for developing extension training materials are to be 
defined, identified, and satisfied is the need for improved 
management evaluation of the extension training materials 
program. Systematic feedback o'l extension training materials to 
developers and managers could be very useful in future 
development efforts and in evaluating and improving overall 
program effectiveness. 
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Unlike the Army correspondence course training^ the Army 
does not routinely receive data on other extension training 
materials usage and effectiveness. Personnel at the schools said 
they receive some feedback on product effectiveness when visiting 
units to validate completed training materials and on usage 
levels during periodic visits to field installations. However, 
this does not constitute systematic feedback, and school 
personnel could not identify specific uses made of or actions 
taken on the usage data. 

The various Army studies, surveys, and audits have shown 
that training materials usage is generally low; these reports 
could serve as a basis for further evaluation of specific 
training materials. The Army does not have criteria for 
evaluating the extent of materials usage or for use in judging 
the relative merits of various usage levels. 

We believe the Army needs to develop such criteria and 
obtain user feedback on individual materials. Furthermore, we 
believe the Army needs to ensure that lessons learned from usage 
of extension training materials previously fielded are considered 
before proceeding with its planned large-scale effort to develop 
new materials, many of which are for new or revised military 
occupational specialties related to new systems under develop- 
ment. In addition to being useful for overall program evalu- 
ation, such feedback would also be useful for determining the 
need to retain and/or revise existing materials in units. 
Currently, the schools periodically review fielded training 
materials and revise them if doctrine or equipment changes. This 
is done without considering actual use of the materials. We 
believe a more complete evaluation loop for training materials 
provided by information on individual materials usage and 
effectiveness would help ensure that training resources are not 
expended unnecessarily. 
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United states general accounting office 

WASHINGTON. D.C. 20548 



NATIONAL SECURITV ANO 
INTENNATIONAL AFFAIRS DIVISION 



JUL 1 i 1984 



The Honorable Delbert L. Spurlock 
The Assistant Secretary of the Army 
(Manpower and Reserve Affairs) 

Dear Mr. Spurlock: 

Subject: Observations on Army's plan to buy an electronic 
video-disc delivery system (GAO/NSIAD-84-1 40 ) 

As part of our current review of the Army's program for 
developing soldier training materials (code 967106), such as 
training extension course lessons , we have inquired about Army 
plans to purchase new training equipment. The purpose of this 
letter is to bring to your attention our concern over whether 
the acquisition plans for video-disc equipment are justified 
based on the low usage of current training materials by soldiers 
in the field. 

BACKGROUND 

The Army has programmed about $140 million for procurement 
of an electronic video-disc delivery system , as well as the- 
development and conversion of training materials to video-disc r 
for fiscal years 1985 through 1990. Life-cycle cost of the 
equij^Tient over the next 20 years is Estimated to range as high 
as $388 million (adjusted for inflation) for the 20,000 units in 
the approved acquisition objective. Over 80 percent of equip- 
ment that the Army plans to buy by 1990 are for use by soldiers 
in the field. According to personnel in the Army Communicative 
Technology Office' — which has initiated the acquisition of the 
new video-di&c system — total equipment units could reach 40,000 
if the Army fields the system worldwide and adapts it to other 
applications such as maintenance and repair. 

Army Communicative Technology Office personnel told us that 
the first 920 equipment units to be purchased are for the Army's 
training schools and that about 17,000 of the remaining 19,080 



This is a joint U.S. Army Materiel Development and Readiness 
Command/U.S. Army Training and Doctrine Command office 
colocated with the Training and Doctrine Command's Army 
Training Support Center at Fort Eustis, Virginia. 
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units will be placed in troop units beginning in fiscal year 
1987. Technology Office personnel also told us that about 
$900^000 will be spent this fiscal year for development and con- 
version of training materials to video-disc and that about $25 
million has been programmed for this purpose for fiscal years 
1985 through 1990. 

According to Communicative Technology Office officials ^ the 
new equipment will reduce the present volume of paper training 
materials and will provide 

--a standard information delivery system with 
high density storage and rapid access capabil- 
ity to replace the paper and other existing 
audio visual delivery systems^ 

— the latest state-of-the-art in information 
delivery r and 

--interactive programmed instructions and simu- 
lation to the user (for example ^ the user 
can automatically reverse the materials for 
replay or skip materials). 

SOLDIERS. USE OF TRAINING MATERIALS 

Our concern about the justification for the new equipment 
and conversion of training materials to video-disc format 
centers around the decision to provide a large amount of 
training equipment to troop units even though the Communicative 
Technology Office has not analyzed soldiers' past usage levels 
of training materials. This concern is heightened by studies 
and other data obtained during our review which indicates that 
soldiers do not use a large portion of available training 
extension course materials. 

According to Communicative Technology Office officials ^ the 
decision regarding the number of equipment units to be placed in 
troop units was based on providing each battalion in both the 
active Army and reserve components with four units (one equip- 
ment unit per company level troop unit) and not on an analysis 
of training materials' usage levels. 

Our review of training materials' usage has been limited to 
an analysis of available data for one major type of extension 
training material--training extension course lessons. According 
to information provided by the Army Training Support Center, 
training extension course lessons represented about 31 percent 
of all extension training materials fielded as of May 1984. 
Although the Army has not established criteria for evaluating 
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the extent of training materials usage, it appears — on the basis 
of limited Army studies — that soldiers do not use these lessons 
extensively. 

A 1979 Army Research Institute study showed that only about 
50 percent of the soldiers surveyed through questionnaires had 
used training extension course materials. The study also showed 
that 82 percent of recorded uses for selected units during a 2- 
month period were directed by commanders. 

More recent Army studies have also disclosed limited use of 
training extension course lessons. Fur example, in 1982 the 
U.S. Army Audit Agency reported that use of training extension 
course lessons at the 5th Infantry Division, Fort Polk, 
Louisiana, was very low. It reported that 

— 52 of 100 soldiers interviewed had never used the 
materials, 21 had never heard of them? 

— only 7 of 43 companies had included the materials 
in their training schedules, and then in only 36 of 
441 available training days? and 

— the average number of soldiers using these mater- 
ials each month was generally very low compared to 
the battalion strength level — from 5 to 22 percent 
for 5 battalions, and 42 percent for the sixth 
battalion reviewed. 

The Axnny Training Support Center is conducting another 
training extension course lesson usage survey which is scheduled 
for completion later this year. 



In conclusion, in view of the lack of a thorough analynis 
of th^ various types of training materials' usage levels, it 
appears that there is an inadequate basis for the Army to 
determine how many equipment units to buy and what types of 
training materials to convert to video-disc. We believe that 
low usage levels may be more indicative of a lack of need for 
the training materials being furnished to units in the field 
than to a deficiency in the type of system used to present the 
material. Therefore, there may be little reason to believe that 
the acquisition of new video-disc equipment will result in 
increased training materials' usage. Our continuing review of 
the Army's program for developing soldier training materials 
will address the question of identified training material needs. 
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We have discussed this concern with personnel in the 
Training Directorate, Deputy Chief of Staff for Operations and 
Plans, Department of the Army? and the Conmunicative Technology 
Office. If you desire, we would be glad to discuss this further 
with you. 

We would appreciate being advised of your views on the 
matters discussed in this letter as well as any actions that you 
may plan to take. We are sending a copy of this letter to the 
Secretary of the Army, 



Sincerely yours. 




Henry W. Connor 
Senior Associate Director 
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THE ASSISTANT SECRETARY OF DEFENSE 



WASHINGTON. D C 20301 



MANPOWER. 
INSTALLATIONS 
AND LOGISTICS 



2 OCT 1984 



Mr. Henry W. Conner 
Senior Associate Director 

National Security and International Affairs Division 
United States General Accounting Office 
Washington, D.C. 20548 

Dear Mr. Conner: 

This is the Department of Defense response to GAG Final 
Report, subject: Observations cn Arm: 's Plan to Buy An 
Electronic Video-Disc Delivery System, No. GAO/NSIAD-84-140, 
dated July 11, 1984 (GAG Code No. 967106), GSD Case No. 6559. 

The GAG Report addressee Ariiy plans to buy an electronic 
video-disc system and to produce extension training materials for 
delivery by video-disc. The report observed that acquisition 
plans for video-disc equipment may not be based on a validated 
need, given low usage of current training materials by soldiers 
in the field. 

Plans for the Electronic Information Delivery Syst*> (BIDS), 
when finalized, will assure that the number of systems >cured 
are compatible with expected usage levels. Hardware del ,ery 
will be synchroiixzed with the distribution of video-disc based 
training materials. This system is programmed over a five-year 
period with the initial procurement of 820 sets in FY 86. The 
largest number of sets (12,880) is to be acquired in FY 90. 
Should BIDS usage prove to be lower than projected after initial 
fielding, procurement levels will be appropriately reduced. The 
DoD position on the findings is enclosed. 

There is every reason to expect that the fielding of BIDS 
will provide the Army with a cost effective means to undertake 
sustainment training, especially for perishable and highly 
technical skills. 



Sincerely, 




Enclosure 
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GAO PINAL REPORT - NO. GAO/NSIAD-84-140 « DATED JULY 11 r 1984 
(GAG CODE NO. 967106) OSD CASE NO. 6559 

••OBSERVATIONS ON ARMY'S PLAN TO BUY AN 
ELECTRONIC VIDEO-DISC DELIVERY SYSTEM" 



* * * * * 

FINDING 



FIND I NG A ; Array Has Programined $140 Million For Procureroent Of 
Electronic Vxdeo-Disc Delivery Systera To Be Used In Training . 
GAO found that the Array has programmed about $140 raillion for 
procurement of an electronic video-disc system^ as well as the 
developraent and conversion of training material to video-disc 
for fiscal years 1985 through 1990. GAO also found that the 
life-cycle cost of the equipment for the next 20 years is 
estiraated to range aj5 high as $388 raillion (adjusted for 
inflation) for the 20^000 units in the acquisition approved 
objective. In addition^ GAO found that over 80 percent of 
equipraent that the Array plans to buy by 1990 are for use by 
soldiers in the field. (p. 9^ GAO Letter Report). 
(See GAG note^ p. 16. ] 

DOD PARTIALLY CONCURS . Array has programmed $100.3 raillion to 
procure approxiraately 20^000 Electronic Inforraation Delivery 
Systera (EIDS) sets over a period of five years starting in Fiscal 
Year 1986. There is $27 raillion programmed for the developraent, 
production and distribution of courseware to be delivered using 
video-disc during the period Fiscal Year 1984 to Fiscal Year 
1990. It is planned to provide approxiraately 80 percent of the 
EIDS equipraent to support distributed horae station training for 
Active Array and Reserve Coraponent soldiers. 

FINDING B . Effect of Video-Disc on Paper Training Materials . 
GAO found that according to Array Communicative Technology Office 
(ACTO) officials, the new video-disc equipraent will reduce the 
present volume of paper training raaterials. In addition^ GAO 
reported ACTO clairas that the video-disc will provide the 
following: 

(1) A standard inforraation delivery systera with high density 
storage and rapid access capability to replace paper and 
other existing audio visual delivery systeras; 

(2) the latest state-of-the-art in information delivery; and 

(3) interactive programmed instruction and simulation to the 
user (for example — the user can automatically reverse the 
raaterials for replay or skip raaterials) . (pp. 9-10^ 
Letter Report) 
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PCD CON C URS , High density storage and rapid access of informa- 
tion capabilities of microprocessor controlled video-disc systems 
offers advantages over some paper based instructional materials 
for selected military applications. Th^'s technology is particu- 
larly useful where large volumes of technical information are 
to be presented r on call^ to explain and demonstrate complex 
functions for operation and maintenance of highly technical 
equipment and weapons systems. While this type system is not 
appropriate for all information delivery applications^ it can 
provide substantial savings over existing more costly methods 
when the courseware is properly designed ^ packaged and dis- 
tributed to a specified target audience. Military Standards are 
being developed to ensure compatibility and interoperability of 
contract or in-house produced courseware with the 
microprocessor/video-disc hardware. With standard poirts to 
connect peripheral equipment such as key boards^ printers^ 
computer discs and a modem^ this system provides the flexibility 
to be tailored for many uses for many years to com^^. 

FINDING C ; Soldiers Use of Training Materials . GhO expressed 
concern that the justification for the new equipment and 
conversion of training materials to video-disc format centers 
around the decision to provide a large amount of training 
equipment to troop units even thoigh the Army , Communicative 
Technology Office has not analyzed soldiers' past usage level of 
training materials. GAO found through studies and other data 
that soldiers do not currently use a large portion of available 
training extension course materials. GAO concluded that in view 
of the lack of a thorough analysis of the various types of 
training material usage levels^ there is an inadequate basis for 
the Army to determine how many equipment units to buy and what 
types of training materials to convert to video-disc. GAO 
further concluded that low usage levels may be more indicative of 
a lack of need for the training materials being furnished to 
units in the field than to a deficiency in the type of system 
used to present the materials. GAO finally concluded, therefore, 
there may be little reason to believe that the acquisition of new 
video-disc equipment will result in increased training materials, 
usage. (pp. 10-1 1 , GAO Final Letter Report) 

POD NON-CONCURS . Analysis of existing training material usage, 
Guch as the Training Extension Course (TEC) , has been ongoing for 
several years by the US Army Training Support Center (USATSC) , 
proponent schools, the Army Research Institute (ARI) and the HQDA 
Staff. User surveys have been conducted through the Army 
Personnel Center ?urvey System. The Lessons Learned have bean 
used to develop plans to improve management, distribution of 
courseware and equipment procurement methods. 
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Usage level is only one of several factors to consider in 
deteriuining requirements for extension training. Other factors 
include the number and types of tasks that are taught during 
initial entry at training centers and schools compared to those 
which must be taught by the Active and Reserve Component organi- 
zations to which individuals are assigned for duty. Tiaininq 
strategy impacts on the selection of the media to be used. The 
media selected is a prime factor in determining the character- 
istics of the presentation equipment to be used. The amount of 
presentation equipment required to accomplish the training 
^^^^''^iK^^ IS determined by the expected usage of the courseware, 
the number and types of Military Occupational Specialties (MOS) 
in using units, the training time and training support resources 

?n?'^^^}^' scheduling of Skill Qualification TestfcsQ?^ 

tor assigned personnel. v-^wj^/ 

Pilot fielding of Interactive video-disc courseware to units 
J?n =f l^lr'^o^l ' acceptance and usage levels. Additionally, 
820 sets of BIDS hardware and interactive video disc courseware 
nHir^L^^'-^^^-^fi^^^'?^ proponent institutional sSSJo!s 
Irl L^n ^? FY 87. only 960 sets of BIDS hardware- 

are programmed for distribution for use by field units and other 
organizations in FY 87. Bxpected usage of lessons a^d ha^dSarl 

• P'^e-determined and closely monitored for instructional 
f;fK^!tr'"^f^ acceptance and product and hardware dis- 

tribution efficiency. Should BIDS csage prove to be lower than 
pro:ected, procurement levels will be Appropriately reduced 

SnS''H?!?"K^?^^!-''"^''^^ perishable skills will be developed 
wni if P"^^ ^y^^^'"' demand supported, 

will be used rather than the push method used for the inital 

distribution of Training Bxtension Courses (TBC) . The Blectronic 
is S^det?" system (BIDS) permits instr^ctlSna! ma^I??a? 

to be delivered by interactive, individually-oaced mode or as an 
instructor-controlled group presentation. ' 

For these reasons, the fielding of BIDS is expected to provide 
Irlin'i^l effective means to undertake sustainment 

training, especially for perishable and highly technical skills 



GAG note: 



rhfLlf f""*""^^^ ^" ^""^^ appendix have been 
Changed to correspond to those in the final 
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THE ASSISTANT SECRETARY OF DEFENSE 

WASHINGTON, D.C. 20301 



1 6 MAY 1985 



Mr. Frank C. Conahan 

Director^ National Security and 

International Affairs Division 
U.S. General Accounting Office 
441 G. Street, NW 
Washington, D.C. 20548 

Dear Mr. Conahan: 

Please find enclosed the Department of Defense (DoD) response 
to the draft GAO report, "Improvenents Needed in the Army's 
Program for Developing Extension Training Materials for Use by 
Soldiers in Field Units", dated March 13, 1985 (GAO Code 967106, 
OSD Code 6559-A) . 

The report accurately identifies a series of related 
weaknesses in Army development and management of extension 
training materials. DoD agrees that these weaknesses could 
result in the creation of training materials which are not used 
sufficiently to justify the expense of their development and 
production. As the report indicates, the Army has taken several 
steps to correct these weaknesses. However, DoD agrees that 
further review and improvement of the orocess is in order to 
assure that all the necessary guidance is in place and that the 
training materials and the training support program are 
functioning properly to meet field needs. There is no question 
that a systematic evaluation and use of che evaluation results is 
essential to cost-effective extension training material 
development, production and use. The attachment to this letter 
discusses each of these issues in greater detail. 

The same requirement for systematic evaluation of training 
materials, hardware, and plans applies to the Army video-disc 
system. The proper point for assessment of the video-disc field 
support program, however, is in FYS9, after training materials 
and equipment become available for field use and evaluation, 
rather than FY87, which is before the materials and hardware are 
available for field use and evaluation. 
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DoD appreciates your assistance in improving training 
support. 

SincereJ-y, 




JinfL Caih 

^uty AMistant S«cr«taryof Oefema 
(Mmpewir, InstalUtjons fc Loginics) 
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GAO DRAFT REPORT 

"Improvements Needed in the Army*s Program tor Developing Extension 
Training Materials for Use by Soldiers in Field Units" 
GAO CODE 967106, OSD CASE 6559-A 
DoD Comments on GAO FINDINGS AND RECOMMENDATIONS 



FINDINGS 

FINDING A; The Army Needs To improve The Process For Developing 
Extension Training Materials . The GAO noted that the U.S. Army 
Training and Doctrine Command (TRADOC) , through its Service 
schools, develops training materials for (1) resident school 
training of soldiers and (2) support of unit and individual 
soldier training — commonly referred to as extension training. 
The GAO also noted that the Army has initiated actions to improve 
its process for developing extension training materials by 
issuing guidance and an ongoing survey of several thousand 
soldiers. The GAO, however, found that TRADOC may be developing 
and revising extension training materials that are not needed or 
may not be used sufficiently to justify their cost. The GAO 
further found that several factors (1) the earlier Army attempt 
to eliminate half of the then existing materials, (2) the con- 
sistent indications of low usage of materials in the field, based 
on Army evaluations, and (3) the 16,000 different materials 
presently available and the thousands of extension training 
materials under development and programmed for future develop- 
ment, at a cost of many millions of dollars, indicate that 
improvements are necessary. The GAO concluded that unless 
improvements are made, TRADOC will continue to develop and revise 
extension training materials for which there is little assurance 
of need. (p.1 Letter and pp. 1, 5, Appendix I, GAO 
Report) 

[See GAO note, p. 24.] 

DoD Concurs . TRADOC has developed a new handbook (February 
1985) for the development of extension training materials. The 
Army is reviewing these procedures for training materials 
development to assure the adequacy of the total process and of 
the resulting materials. 

FINDING B ; Low Use Of Training Materials Raises Questions About 
The Need For The Materials" The GAO found that data is not 
routinely collected on extension training materials usage; 
however, the Army has expressed concern over the large numbers of 
extension training materials, and several Army studies and audits 
conducted' since 1979 have classified usage as low for the 
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Training Extension Course (TEC) lessons. The GAO further found 
that while the indicated low usage levels can raise questions 
about the need for these extension training mater ials^ such low 
use cannot be attributed solely to a lack of need for training or 
for certain types of training materials ^ as none of the studies/ 
audits specifically addressed the need for these extension 
training lessons or the reasons for the usage levels identified. 
However r the GAO concluded that the consistent results of the 
studies r surveys r and audits over the last 5 years showing low 
usage indicate the Army needs to (1) define what acceptable usage 
levels are^ (2) obtain feedback on usage levels and reasons for 
the usage^ and (3) use this information as appropriate to improve 
the method for deciding to develop materials and/or ensure 
necessary use by the soldiers. (pp. 1-2^ Letter and pp. 3-5^ 
Appendix 1^ GAO Report) 

DoD Concurs . As a consequence of the Army review and improvement 
of its training materials development procedures ^ the Army must 
produce criteria for acceptable usage levels ^ include 
requirements to assess usage levels and reasons for the usage 
levels r and assure appropriate use of this feedback in improving 
the materials and their use. 

FINDING C : Decisions To Develop Extension Training Materials 
Should Consider Actual Field Needs . Even though the December 
1982 TRADOC guidance specified that the materials must be needed 
and wanted by field units ^ the GAO found that TRADOC' s schools 
plan to spend millions of dollars developing thousands of 
extension training materials on the basis of a specified task 
analysis which does not specifically query field units regarding 
their need for these materials. The GAO further found that even 
though the TRADOC guidance states that an extension training 
material "should be developed only if it is required and meets 
the stated needs of field units^" the procedure does not specify 
how the schools are to determine field needs and does not define 
an actual or a stated field need. The GAO also found that^ 
except for those few instances when field units asked for train- 
ing mater ialSy the schools could not provide any documentation 
from trip reports or otherwise showing where actual training 
needs of field units had been identified from these sources. The 
GAO concluded that without field input on training needs ^ 
materials could be developed which are not needed or do not meet 
user needs. The GAO further concluded that the Army needs to 
define what the determination of field needs means ^ and establish 
procedures for obtaining this input so the schools can consider 
field needs when making decisions on developing training 
materials for them. (pp. 2-3 Letter and pp. 6-7 r Appendix I GAO 
Report) 

DoD Concurs . As a consequence of the Army review and improvement 
of its training materials development procedures^ the Army must 
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clarify what field needs are^ review existing procedures for 
adequacy of obtaining field input on training materials needs, 
and design and develop materials based on that input. 

FINDING D : laproved Prograa Evaluation is Needed For The Exten- 
sion Training Materials ProqraM , The GAO found that the Army 
does not routinely receive data on other extension training 
materials usage and effectiveness. The GAO repoirted that 
personnel at the schools stated they received some feedback on 
product effectiveness when visiting units; however ^ the GAO found 
that this does not constitute systematic feedback^ and school 
personnel could not identify specific uses made of or actions 
taken on the usage data. The GAO concluded that there is a need 
for improved management evaluation of the training materials 
program and that systematic feedback on extension training 
materials to developers and managers could be very useful in 
future development efforts and in evaluating and improving the 
overall program effectiveness. GAO also concluded that Army 
needs to assure that lessons learned from usage of extension 
training materials previously fielded are considered before 
proceeding with its planned large scale effort to develop new 
mater ialSy many of which are for new or revised occupational 
specialties related to systems under development. (p. 3 Letter 
and pp. 7-8^ Appendix l^ GAO Report) 

DoD Concurs . The Army must obtain feedback on extension training 
materials as a component of its systematic assessment of the 
effectiveness of the total training program. As a consequence of 
the Army review and improvement of its training materials 
development procedures^ the Army will assure that guidance on 
training materials evaluation is in place^ that the feedback is 
considered in revising and developing training materials^ and 
that program effectiveness is monitored. 

FINDING E ; Further Evaluation Is Needed Before The Aray Buys And 
Fields New Video-Disc Equipaent > The GAO reported that the Army 
plans to procure a new video-disc electronic information delivery 
system to replace existing paper and audiovisual delivery systems 
and to develop and convert training materials to video-disc for 
about $127.3 million in fiscal years 1984-1990. The GAO noted in 
its July llr 1984 letter report to the Assistant Secretary of the 
Army for Manpower and Reserve Affairs (OSD Case 6559) expressing 
concerns about these planned expenditures ^ as over 80 percent of 
the 20^000 units were to be used by soldiers in the field and 
justification for the purchases did not include an analysis of 
past extension training materials usage. The GAO also noted that 
the DoD response to its letter report agreed that Army analysis 
of extension training material should be continuous. GAO con- 
cluded that the actions planned to further evaluate the need for 
large numbers of the video-disc equipment were steps in the right 
direction. However^ GAO also concluded that the Army needs (1) 
to proceed cautiously and take advantage of the time available 



21 

31 



APPENDIX IV 



APPENDIX IV 



prior to 1987^ when fielding of the equipment is planned^ to 
perform further field testing of the new equipment and (2) to 
incorporate the results of the needed management improvements in 
the program for developing extension training materials. The GAO 
further concluded that these results should be incorporated in 
the reevaluation of the need for large numbers of video-disc 
equipment. (pp. 3-5 Letter, pp. 9-16, Appendix I, Appendix II 
and Appendix III, GAO Report) 

DoD Concurs . The Army will use the time in the interim to 
prepare for effective use of the technology and to develop 
effective processes to stimulate cost effective uses of both the 
new equipment and related training materials. The video-disc 
procurement will not take place until FY 1986, the courseware 
development will take as long as a year, and the first units will 
not be provided to field units until FY 1988. There will be a 
need for some training and development of related training 
materials prior to the actual fielding. The Army's evaluation of 
these needs will be thorough and continuous through the time of 
actual procurement. A full reevaluation of the program and 
training material needs, however, can not be conducted until FY 
1989, after video-disc equipment is fielded and actual usage is 
determined. Any adjustment to either the equipment or training 
material purchases then can be accomplished beginning in FY 1989 
or FY 1990, depending upon when the reevaluation can be 
completed. 

RBCQMHENDATIONS 

RECOMMENDATION 1 . The GAO recommended chat the Secretary of the 
Army direct the Commander, US Army Training and Doctrine Command 
to develop criteria for what is to be considered an actual or a 
stated field need tor extension training materials to include 
what are acceptable usage levels for the materials. (p. 5 
Letter, GAO Report) 

POP Concurs . HQ TRADOC will be tasked immediately to begin 
developing criteria for identifying field needs and defining 
acceptable usage levels, along with appropriate procedures for 
improving usage levels, adjusting management or fielding 
practices to solve the problem, and/or revising the materials to 
meet the problem. 

RECOMMENDATION 2 . The GAO recommended that the Secretary of the 
Army direct the Commander, U.S. Army Training and Doctrine 
Command to delineate procedures in TRADOC guidance which specify 
(1) how school should obtain field input on actual needs and (2) 
how this input is to be used in developing materials. (p. 5 
Letter, GAO Report) 

POP Concurs . HQ TRADOC will be tasked to review and improve its 
procedures for training materials development, assure the 
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adequacy of the total procf=^sSy and assess periodically the 
adequacy of the resulting materials. This training materials 
development process will include procedures for obtaining field 
input on training materials needs and guidance on how this input 
will be used in developing materials. This review and improve- 
ment should be complete in 1986. 

RECOMKBNDATIOH 3 . The GAO recommended that the Secretary of the 
Army direct the Commander ^ U.S. Army Training and Doctrine 
Command to obtain feedback on individual training materials usage 
and effectiveness for the purposes of incorporating lessons- 
learned into the development of requirements for new extension 
training materials and determining the need to revise existing 
materials. (p. 5 Letter ^ GAO Report) 

DoD Concurs > TRADOC will be tasked ^ as an element of its review 
of the process of identifying and meeting field training support 
needs, to assure the systematic checking of training materials 
use and effectiveness. TRADOC will also be tasked with the 
requirement to periodically assess the effectiveness of the 
feedback procedure in assuring materials and program effective- 
ness and efficiency. 

RECCMMKBNDATIOH 4 , The GAO recommended that the Secretary of the 
Army assure the Army's reevaluation of the planned 1987 procure- 
ment of new video-disc equipment units for field use includes 
consideration of (1) the usage levels for extension training 
materials and the reasons for such usage, and (2) the status of 
other management improvements needed in the program for develop- 
ing such materials, such as the development of criteria defining 
acceptable usage levels and improvement in the criteria and 
procedures for determining field needs for extension training 
• materials. (p. 5 Lettir, GAO Report) 

DoD Partially Concurs , The evaluation of video-disc materials, 
hardware, and applications in institutional training and in 
extension training will be thorough and continuous. The Army 
will use the time in the interim to prepare for effective use of 
the technology and develop effective processes to stimulate cost 
effective uses. Specifically, the Army will refine its video- 
disc development process during the fielding to the schools in FY 
1986 and 1987 and develop written guidance for that process, to 
include the lessons learned from the evaluation of extension 
training materials. The Army also will develop detailed fielding 
plans for che video disc system that speak to resident and ex- 
tension training material development, management, and use 
strategies. 

Since video-disc procurement will not take place until FY 1986 
and field units will not receive the equipment until FY 1988, 
field use of exttnsion training video-disc materials can not be 
used to reevaluate the FY 1987 procurement as recommended by the 
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GAO. In addition, equipment provided to field units in FY 1988 
will be contingent on courseware availability. However, an 
estimated 800 units will be available for FY 1988 field 
evaluation of usage levels and field effectiveness. The Army, 
therefore, will reassess its procurement plans in FY 1989 or FY 
1990. (See response to Finding E.) 



GAO note: Page references in this appendix have been changed 
to correspond to those in the final report. 
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